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Abstract 

Cloud services have turned remote computation into a commodity and enable convenient online 
collaboration. However, they require that clients fully trust the service provider in terms of con¬ 
fidentiality, integrity, and availability. Towards reducing this dependency, this paper introduces a 
protocol for verification of integrity and consistency for cloud object storage (VICOS), which en¬ 
ables a group of mutually trusting clients to detect data-integrity and consistency violations for a 
cloud object-storage service. It aims at services where multiple clients cooperate on data stored 
remotely on a potentially misbehaving service. VICOS enforces the consistency notion of fork-line- 
arizability, supports wait-free client semantics for most operations, and reduces the computation and 
communication overhead compared to previous protocols. VICOS is based in a generic way on any 
authenticated data structure. Moreover, its operations cover the hierarchical name space of a cloud 
object store, supporting a real-world interface and not only a simplistic abstraction. A prototype of 
VICOS that works with the key-value store interface of commodity cloud storage services has been 
implemented, and an evaluation demonstrates its advantage compared to existing systems. 


1 Introduction 

More and more data is outsourced to the cloud, and collaborating on a shared resource using cloud 
services has become easier than ever. Programmers work together on online source code repositories, 
global project teams produce technical deliverables, and friends share then - photo albums. Nevertheless, 
the clients need to trust the cloud provider as rely on it for the confidentiality and correctness of their 
data. Encryption may preserve the confidentiality of data but cannot prevent inadvertent or malicious 
data modifications. This work shows how to protect the integrity and consistency of data on an untrusted 
cloud storage service accessed by multiple clients. 

With a single client only, the client may locally keep a short cryptographic hash value of the out¬ 
sourced data. Later, this can be used to verify the integrity of the data returned by the cloud storage 
service. However, with multiple disconnected clients, no common synchronization, and no communica¬ 
tion among the clients, neither hashing nor digital signatures are sufficient by themselves. The reason is 
that a malicious or Byzantine server may violate the consistency of the data, for example, by reordering 
or omitting properly authenticated operations, so that the views of the storage state at different clients 

* A short version of this work appears in the proceedings of SYSTOR 2015. DOI: 10.1145/2757667.2757681 
^Work done at IBM Research -Zurich 
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diverge. A malicious cloud server may pretend to one set of clients that some operations by others sim¬ 
ply did not occur. In other words, freshness can be violated and the clients cannot detect such replay 
attacks until they communicate directly. The problem is particularly relevant in cryptographic online 
voting and for web certificate transparency lf22ll . 

The strongest achievable notion of consistency in this multi-client model is captured by fork-line - 
arizability, introduced by Mazieres and Shasha li26l . A consistency and integrity verification protocol 
may guarantee this notion by adding condensed data about the causal evolution of the client’s views into 
their interaction with the server. This ensures that if the server creates only a single discrepancy between 
the views of two clients, these clients may never observe operations of each other afterwards. In other 
words, if the server ever lies to some clients and these clients communicate later, they will immediately 
discover the violation. Hence, with only one check they can verify a large number of past operations. 

The SUNDR system ll23l pioneered fork-linearizable consistency and demonstrated a network file 
system protected by a hash tree of every user’s files. SUNDR uses an expensive protocol, requiring 
messages of size Q(n 2 ) for n clients l iTOl l . Like other systems providing fork-linearizability, it suffers 
from the inherent limitation that sometimes, even with a correct server, clients have to block and cannot 
proceed with their next operation because other clients are concurrently executing operations lllOll . 

In order to prevent blocking, FAUST [8]|, SPORC lfl3l . and Venus lf30l relax their guarantees to 
weak fork-linearizability, which establishes consistency only eventually, after further operations occur; 
this is not desirable because a client may only know later that a protocol output was not correct. SPORC 
and the related Blind Stone Tablet (BST) Il34ll protocol shift the maintenance of state to the clients, 
such that the server is merely responsible for coordination; every client holds a complete copy of the 
system’s state. This contradicts the goal of outsourcing data to the cloud. Another way to avoid blocking 
is explored by BST and COP they let any commuting operations proceed immediately. 

In this paper, we present VICOS, a verification protocol for the integrity and consistency of cloud ob¬ 
ject storage, which overcomes these limitations and demonstrates a complete practical system. VICOS 
supports the optimal consistency notion of fork-linearizability, provides wait-free semantics for all com¬ 
patible client operations, and has smaller overhead than previous protocols. The notion of compatible 
operations, as introduced here, generalizes the progress condition over commuting operations, which are 
considered in past work. Informally, two operations o and uj are compatible if executing o before u does 
not influence the behavior of u (neither its effects nor its output value). Conceptually, VICOS is based 
on abstract authenticated data structures in a modular way. Moreover, it unifies the two different lines 
of work on this problem so far, namely, the untrusted storage protocols fl23l llOl HI that feature remote 
state and are based on vector clocks, and the remote service verification protocols j34l [T3| |91, which 
create local copies of the state and use hash chains. In particular, VICOS maintains state remotely, at 
the server, but uses hash chains for consistency verification. Furthermore, VICOS incurs only a con¬ 
stant communication overhead per operation, independent of the number of clients, whereas FAUST and 
Venus require vector clocks of size Q(n) with n clients. 

We have implemented and evaluated VICOS with a commodity cloud object store; the results 
demonstrate that the overhead for distributed, multi-client integrity and consistency verification is low. 
VICOS protects a complete cloud storage service (assuming it has atomic semantics), spanning many 
objects and offering read, write, delete, and directory listing operations; this stands in contrast to 
Venus 1301 , which only provided consistency for a single data object. Furthermore, VICOS notifies 
the clients whenever the integrity or consistency is violated but does not address recovery operations. 

1.1 Contributions 

This work makes the following contributions towards ensuring integrity for data stored on untrusted 
cloud providers: 
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• A novel abstract protocol to verify the integrity and consistency of a generic service based on an 
authenticated data structure lf32l . which ensures fork-linearizability, supports wait-free semantics 
for compatible operations, and incurs only constant communication overhead. This protocol also 
generalizes authenticated data structures to multiple writers. 

• An instantiation of this protocol for a commodity cloud object store, called VICOS. It represents 
the first integrity protection protocol with all of the above features for the standard operations of 
a cloud object store. 

• An implementation and evaluation of VICOS using COSBench, demonstrating its practicality. In 
particular, the overhead of VICOS for integrity protection remains acceptable small with moderate 
concurrency and increases slightly when many clients access the same data concurrently. The 
prototype is available open-source. 

1.2 Organization 

The paper continues with introducing the model and defines our notion of an authenticated data structure 
(ADS). Sec. [3] presents the abstract integrity protocol for ADS and discusses its properties. VICOS is 
introduced in Sec.[4]and the evaluation appears in Sec. [6] Related work is discussed in Sec. [7] 

2 Model 

We consider an asynchronous distributed system with n mutually trusting clients C \..... C n and a 
server S. The communication between the server and the clients is reliable and respects first-in first-out 
(FIFO) semantics. Clients cannot communicate with each other. A protocol P specifies the behavior of 
the clients and the server. All clients are correct and hence follow P\ in particular, they do not crash 
(although crashes could be tolerated with extra measures in the protocols). The server is either correct 
and follows P or Byzantine, deviating arbitrarily from P. 

The clients invoke operations of a stateful functionality F, implementing a set of deterministic 
operations; F defines a response and a state change for every operation. We use the standard notions 
of executions, histories, sequential histories, real-time order, concurrency, and well-formed executions 
from the literature 0. In particular, every operation in an execution is represented by an invocation event 
and a response event. We extend F with a special return value ABORT that allows an operation to abort 
without taking effect, which may be used when concurrent operations would cause it to block fl24l (9ll. 

2.1 Consistency properties 

When S is correct, the protocols should provide the standard notion of linearizability lfl9l with respect 
to F. It requires that an execution of operations by all clients together is equivalent to an imaginary se¬ 
quential execution of F. More precisely, for an execution to be linearizable, all invocation and response 
events occuring at all clients can be permuted into one totally ordered sequence, where (1) every oper¬ 
ation invocation is followed immediately by the corresponding response; (2) all operations are correct 
according to the specification of F\ and (3) the operations in the sequence respect the real-time order 
among operations observed in the execution. 

Fork-linearizability ||26l ITOl relaxes this common global view and permits that the clients observe 
an execution that may split into multiple linearizable “forks,” which must never join again. More pre¬ 
cisely, an execution is fork-linearizable when every client observes a linearizable history (containing all 
operations that the client executes itself) and for any operation observed by multiple clients, the history 
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of events occurring before the operation is the same at those clients. This implies that if the views of the 
execution at two clients ever diverge, they cannot observe each other’s operations any more, and makes 
it easy to spot consistency violations by the server. 

Furthermore, we recall the concept of a fork-linearizable Byzantine emulation iflOl , It requires that 
our protocol among the clients and the Byzantine server satisfies two conditions: When the server is 
correct, then the service is linearizable; otherwise, it is still fork-linearizable. Finally, our protocol may 
only abort (by returning ABORT) if there is some reason for this; in other words, when the clients execute 
operations sequentially, then no client ever aborts. 

2.2 Cryptographic primitives 

Our protocols use a cryptographic hash function and digital signature schemes for protecting data against 
modification. We model them in an idealized way, as if implemented by a distributed oracle Q. 

A cryptographic hash function hash maps a bit string x of arbitrary length to a short, unique hash 
value h. Its implementation is deterministic and maintains a list L of all x that have been queried so 
far. When the invocation contains x E L, then hash responds with the index of x in L; otherwise, hash 
appends x to L and returns its index. This ideal implementation models only collision resistance, i.e., 
that it is not feasible to find two different values x\ and X 2 such that hash(x i) = hash(x 2 ). 

A digital signature scheme as used here provides two functions, sign { and verify to ensure the 
authenticity of a message created by a known client. The scheme works as follows: A client Cj in¬ 
vokes signfm ) with a message m as argument and obtains a signature d> E {0,1}* with the response. 
Only client Q can invoke sign { . When m and 0 are sent to another client, that client may verify the 
integrity of m. It invokes verify-( 0 , m) and obtains TRUE as a response if and only if C% has executed 
sign,- (rri) ; otherwise, verifyff, m ) returns FALSE. Every client, as well as S, may invoke verify. 

2.3 Authenticated data structures 

This section defines the model of authenticated data structures (ADS) used here. Authenticated data 
structures ||28l 1251 [321 are a well-known tool for verifying operations and their results over data out¬ 
sourced to untrusted servers. Popular instantiations rely on Merkle hash trees or other hierarchical 
authenticated search structures lfT71l27ll . 

We model an ADS for an arbitrary deterministic functionality F. Departing from the literature 
on ADSs, we eliminate the special role of the single writer or “source” and let any client perform 
update operations; likewise, we unify queries and updates into one type of operation from a set O. 
Operations may contain arguments according to F, but these are subsumed into the different o E O. 
The functionality specifies a state s E S, which will be maintained by the server, starting with an 
initial state sq- For example, this may include all data stored on a cloud storage service. Given s, 
applying an operation o of F means to compute (s', r) 0 - F(s, o ), resulting in a new state s' E S and a 
response r E 72. 

Operations are executed by the clients and formally described by an invocation event and a response 
event occurring at the client. In order to verify the responses of S, a client stores an authenticator a, a 
short value also called a digest. Initially, the authenticator is a special value oq. 

For executing an operation o, the client invokes algorithm query F on S, 

(r, 0 - o ) <- query F (s,o), 

producing a response r and auxiliary data o Q for o; the latter may serve as a proof for the validity of the 
response. Then the client locally performs an operation authexecF to validate the response on the basis 
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of the authenticator. From this the client obtains an output 

( a',a' 0 ,v) -c— authexec F (o,a,r,o 0 ). 

Here a' and o' 0 are the updated authenticator and auxiliary data, respectively, and v E {false, true} 
denotes a Boolean verification value that tells the client whether the response r from S is valid. 

The client should then send o' D back to S, so that the server may actually execute o and update its 
state from s to s' by running refresh F as 

s' «— refresh F (s,o,cr' 0 ). 

Note we may also consider these operations for sequences of operations. 

An ADS ll28l [25l l32ll is a special case of this formalization, in which the operations O can be 
partitioned into update-operations U and query-operations Q. Update operations generate no response, 
i.e., F(s, u) = (s' , _L) for all u E U and queries do not change the state, that is, F(s , q) = (s, r) for all 
q £ Q- 

Furthermore, an ADS may contain initialization and key-generation routines and all algorithms may 
take public and private keys as inputs in addition. For simplicity, and because our ADS implementations 
are unkeyed, we omit them here. 

An ADS satisfies correctness and security. Consider the execution of a sequence (oi,..., o m ) of 
operations, F(so, ( 01 ,..., o m )), which means to compute ( Sj,rj ) <— F(sj- 1 , Oj) for j = 1,... ,m. A 
proper authenticated execution of (cq, ..., o m ) computes the steps 

(' r j,<r 0 ,j) query F (sj-i,Oj) 

(dj, cr' 0ij ,Vj) E- authexec F (oj,a,j-i,rj, o Q j ) 

Sj E- refresh F (sj-i,Oj,a' 0 j). 


such that Vj = TRUE, for j = 1 ,,m. 

An ADS is correct if the proper authenticated execution of any operation sequence ( 01 ,..., o m ) 
outputs state s m and response r m such that ( s m ,r m ) = F(sq, (oi, ..., o m )). 

Furthermore, an ADS must be secure against an adversary A that tries to forge a response and 
auxiliary data that are considered valid by a client. More precisely, A adaptively determines an operation 
sequence (o\ ,..., o m ) , which is taken through a proper authenticated execution by a challenger; at every 
step, A obtains a,j and o' Q ■ and then determines o ]+ \ and whether the execution continues. Finally, after 
obtaining a m and s m , A outputs an operation o*, a response r* , and a value cr*. The ADS is secure if 
no A succeeds in creating o*, r*, s* m , and o* such that 

true) = authexec F (o* ,a m ,r* ,ol ) 

but F(s m , o*) = (s m+ 1 , r m+ 1 ) with r m+ i A r *■ (The “don’t-care” symbol • in the tuple indicates that 
only a subset of the tuple elements are needed.) In other words, A cannot find any o* executed on s m 
and forge a response r* and a proof for o* that is accepted by the client, unless the response is correct 
according to F and r* = r m+ \. 

Note that this formalization represents an “idealized” security notion. It is easy to formulate an 
equivalent computational security condition using the language of modern cryptology Ifl5l . This model 
subsumes the one of Cachin |[6) and generalizes ADS lf32l to multiple writing clients, where here the 
authenticator is implicitly synchronized among the clients. 
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2.4 Compatible operations 

Our protocol takes advantage of compatible operations that permit “concurrent” execution without com¬ 
promising the goal of ensuring fork-linearizability to the clients. An operation o' is compatible with 
another operation o (in a state s) if the presence of o' before o does not influence the return value of o 
(in s). Compatible operations can be executed without blocking; this improves the throughput com¬ 
pared to earlier protocols, in particular with respect to COP which considered the stronger notion of 
commutative operations. 

Formally, we say an operation sequence co is compatible with an operation o in a state s whenever 
the responses of o remain the same regardless of whether co executed before o. Hence, with 

( s',r ) <— F(s,co ); ( s",p ) F(s',o)\ and 

(t',q) -e- F(s,o ) 

it holds p = q. Moreover, we say that co is compatible with o if and only if a; is compatible with o in all 
states s E S of F. 

In algorithms we use a function compatible F , which takes co and o as inputs and returns TRUE if and 
only if co and o are compatible. 

In the terminology of the database literature lf33l , two operations are compatible if and only if they 
do not exhibit a “write-read” conflict, which is also known as a “dirty read.” 

Note that compatible operations are not necessarily commutative, but commutative operations are 
always compatible. For instance, a query operation q is compatible with an update u in any state; but 
when q returns data modified by u, then q and u do not commute. 

3 The ADS integrity protocol (AIP) 

This section introduces the ADS integrity protocol (AIP), a generic protocol to verify the integrity and 
consistency for any authenticated data structure (ADS) operated by a remote untrusted server. AIP 
extends and improves upon the commutative-operation verification protocol (COP) and its authenticated 
variant (ACOP) of Cachin and Ohrimenko j9}. Sec.[4]shows how to instantiate AIP with an authenticated 
dictionary for protecting cloud storage; the result forms the core of VICOS. 

3.1 Overview 

The processing of one operation in AIP is structured into an active and a passive phase, as shown 
in Fig. [T] The active phase begins when the client invokes an operation and ends when the client 
completes it and outputs a response; this takes one message roundtrip between the client and the server. 
Different from past protocols, the client stays further involved with processing authentication data for 
this operation during the passive phase, which is decoupled from the execution of further operations. 

More precisely, when client Ci invokes an operation o £ O, it sends a signed INVOKE message 
carrying o to the server S. The server assigns a global sequence number (t) to o and responds with a 
REPLY message containing a list of pending operations, the response, an authenticator, and auxiliary 
data needed by the client for verification. Operations are pending (for o) because they have been started 
by other clients and S has ordered them before o, but S has not yet finished processing them. We 
distinguish between pending-other operations, which have been invoked by other clients, and pending- 
self operations, which C l has executed before o. 

After receiving the REPLY message, the client checks its content. In particular - , if the pending-other 
operations are compatible with o, then C t verifies the pending-self operations including o with the help 
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Figure 1 . Protocol messages in AIP. 


of the authenticator; if they are correct, C t proceeds and outputs the response immediately. Along 
the way Ci verifies that all data received from S satisfies conditions to ensure fork-linearizability. An 
operation that terminates like this is called successful, alternatively, when the pending-other operations 
are not compatible with o, then o aborts. In this case, C, returns the symbol ABORT. In any case, 
the client subsequently commits o and sends a signed COMMIT message to S (note that also aborted 
operations are committed in that sense). This step terminates the active phase of the operation. The 
client may now invoke the next operation or retry o if it was aborted. 

Processing of o continues with the passive phase. At some (later) time, as soon as the operation 
immediately preceding o in the assigned order has terminated its own passive phase, S sends an UPDATE- 
AUTH message with auxiliary data and the authenticator of the preceding operation to Ci. When C, 
receives this, it validates the message content and verifies the execution of o unless o had been aborted. 
Using the methods of the ADS, the client now computes and signs a new authenticator that it sends 
to S in a COMMIT- AUTH message. We say that C, authenticates o at this time. When S receives this 
message, then it applies o by executing it on the state and stores the corresponding authenticator; this 
completes the passive phase of o. 

Note that the server may receive COMMIT messages in an order that differs from the one of the glob¬ 
ally assigned sequence numbers due to asynchrony. Still, the authentication steps in the passive phases 
of the different operations must proceed according to the assigned operation order. For this reason, the 
server maintains a second sequence number (6), which indicates the last authenticated operation that the 
server has applied to its state. Hence, S buffers the incoming COMMIT messages and runs the passive 
phases sequentially in the assigned order. 

For ensuring consistency, every client needs to know about all operations that the server has exe¬ 
cuted. Therefore, when S responds to the invocation of an operation by Ci, it includes in the REPLY 
message a summary (including the corresponding signatures) of all those operations that C % has missed 
since it last executed an operation. Prior to committing o, the client verities these operations and thereby 
clears them. 

3.2 Notation 

The protocol is shown in Alg. 00 and formulated reactively. The clients and the server are state ma¬ 
chines whose actions are triggered by events such as receiving messages. An ordered list with elements 
ei, 62 , • • ■, efc is denoted by E = (ei, e 2 ,..., e*,); the element with index j may be accessed as E\j]. 
We also use maps that operate as associative arrays and store values under unique keys. A value v is 
stored in a map H by assigning it to a key k, denoted by H[k] <— v; for non-assigned keys, the map 
returns _L. The symbol || denotes the concatenation of bit strings. The assert statement, parameterized 
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by a condition, catches an error and immediately terminates the protocol when the condition is false. 
Clients use this to signal that the server misbehaved. 

3.3 Data structures 

This section describes the data structures maintained by every client and by the server. For simplicity, 
the pseudo code does not describe garbage collection, but we note where this is possible. 

Every client Ci (Alg. |T]) stores the sequence number of its last cleared operation in a variable c. The 
hash chain H represents the condensed view that Ci has of the sequence of all operations. It is computed 
over the sequence of all applied operations and the sequence of pending operations announced by S. 
Formally, H is a map indexed by operation sequence number; an entry H[l\ is equal to hash(H[l — 
l]|H|Z||j) when the Cth operation o is executed by Cj, with H[ 0] = NULL. Variable Z is a map 
that represents the status (SUCCESS or ABORT) of every operation, according to the result of the test 
for compatibility. The client only needs the entries in H and Z with indices greater than c and may 
garbage-collect older entries. Finally, Ci uses a variable u that is set to o whenever Ci has invoked 
operation o but not yet completed it; otherwise u is _L. 

The server (Alg. [3]) maintains the sequence number of the most recently invoked operation in a 
counter t. In addition to that, the counter b contains the sequence number of the most recently applied 
operation and governs the authentication of operations in the passive phase. Every invoked operation 
is stored in a map I and every committed operation in a map O; both maps are indexed by sequence 
number. The server only needs the entries in I with sequence numbers greater than b. An entry in O, at 
a sequence number b or greater, has to be stored until every client has committed some operation with 
a higher sequence number and may be removed later. Most importantly, the server keeps the state s of 
the ADS for F, which reflects all successful operations up to sequence number b. In contrast to COP @ 
and SPORC lfl3l . where every client maintains a complete copy of the state, here only the server stores 
that state. Moreover, S stores the authenticator for every operation in a map A indexed by sequence 
number. 

3.4 The protocol in detail 

This section describes the ADS integrity protocol (AIP) as shown in Alg. |T]-[3] AIP is parameterized by 
an ADS and a functionality F that specifies its operations through query F , authexecp, and refresh p. The 
client invokes AIP with an ADS-operation o by calling aip-invoke(o)', it completes when AIP executes 
return at the end of the handler for the REPLY message. This ends the active phase of AIP, and the 
passive phase continues asynchronously in the background. 

3.4.1 Active phase 

When client Ci invokes an operation o, it computes an iNVOKE-signature t over o and this proves to 
other clients that Ci has invoked o. Then Ci stores o in u and sends an INVOKE message with o and r to 
the server. 

Upon receiving an INVOKE message with o, the server increments the sequence number t, assigns 
it to o, and assembles the REPLY message for Ci. First, S stores o and the accompanying signature 
in /[£]; the value t is also called the position of o. The pending operations for o, assigned to uj, are 
found in I[b + 1],..., I[t], i.e., starting with the oldest non-applied operation, and include o. In order to 
compute the response and the auxiliary data for o from the correct state, the server must then extract the 
successful pending-self operations p of Ci, using the following method: 


function separate-pending(i, w) 

h <- <); 7 M) 

for k = 1,. .., length(uj) do 

(o'r,j) u[k] 

if j = i then 

if k = length(ui) V status of o' is SUCCESS then 

/it-fio (o') // see text how to get status of o' 

else if j ^ i then 

74-70 (o') 

return (p, 7) 

This method is common to the server and the clients. Note that p includes the current operation 
(which appears at the end of u) but not the aborted operations of C t . The server finds the status of 
a pending-self operation o' of C, in O h + k] (except for o itself, obviously) because C r has already 
committed o' prior to invoking o and because the messages between C, and S are FIFO-ordered. On the 
other hand, C, retrieves the status of o' from Z[b + k\. 

Then S computes the response r and auxiliary data <j 0 by calling query F (s, p) from the ADS for F; 
the response therefore takes into account the state reached after the successful pending-self operations 
of Ci but excludes any pending-other operations present in m. Flowever, the client will only execute o 
and output r when 7 is compatible with o and, therefore, Ci is guaranteed a view in which the operations 
of 7 occur after o. This will ensure fork-linearizability. The REPLY message to C, also includes A[b\ 
containing the authenticator and its AUTH-signature, for the operation with sequence number b. The 
client passes these to authexecF of p for verifying the correctness of the response. Furthermore, the RE¬ 
PLY message contains 6 , the list of all operations that have been authenticated since Ci s last operation. 

In particular, when c is the sequence number from the INVOKE message, 5 contains the operations at 
sequence numbers c + 1,..., 6; when c = b, however, 5 contains still O [b ]. 

After receiving the REPLY message from S, the client ( 1 ) processes and clears the authenticated 
operations in 5, (2) verifies the pending operations in to, and (3) verities that r is the correct response 
for o. These steps are explained next. 

For verifying and processing 5 and the last signed authenticator in a, client C) calls a function 
check-view and verities and/or extends the hash chain for every operation and verities the corresponding 
COMMlT-signature. In particular, this ensures for any operation which has been pending for Ci and must 
be cleared, that the same operation was also authenticated by its originator. Finally, C, also checks the 
AUTH-signature on the authenticator a, which is contained in a. If successful, all operations in 6 are 
cleared and Ci s operation counter c is advanced to the position of the last operation in 5. (The check 
for b = c ensures that 5 contains at least one operation at position c.) 

The client continues in check-pending by verifying that the pending operations are announced cor¬ 
rectly: for every operation in oj, it determines the sequence number l, verities the corresponding IN- 
VOKE-signature r, and checks the hash chain entry //[/]. If there is no entry in H for l, then C, com¬ 
putes the new entry from o, l, j, and H\l — 1 ]; otherwise, C r verifies the that existing entry matches the 
expected value. If this validation succeeds, it means the operation is consistent with a pending operation 
sent previously by S. After iterating through the pending operations, the client checks also that the last 
operation in in is indeed its own current operation o. 

Next, Ci invokes separate-pending to extract p and 7 from 00 (see earlier). Then, C t checks whether 
7 is compatible with u (the last invoked operation). If yes, C % calls the ADS operation authexec p(/z, a, r, o 0 ) 
for verifying that applying the operations in p yields r as the response (recall that p includes o at the 
end). The goal of this step is only to check the correctness of the response, and the authenticator and aux¬ 
iliary data output by authexecp are ignored. Finally, C, commits o by generating a COMMlT-signature 
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Algorithm 1 ADS integrity protocol (AIP) for client C, 

state 

c G No: sequence number of last cleared operation, initially 0 
H : No —» { 0 , 1 }*: the hash chain, initially only f?[ 0 ] = _L 
Z : No — > Z\ status map, initially empty 
u £ O U {!}: current operation or _L if none, initially _L 

function aip-invoke(o) 

u t— o 

T G- sign^INVOKElloll*) 

send message [INVOKE, o, t, c] to S 

upon receiving message [REPLY, 5 , 6, a, w, t, r, 07] from S do 
(a, a 
check-view( 6 , b, a, ip) 
check-pending(ui) 

(p, 7) G- separate-pending(i , ui) 
t G- b + length(w) 
if compatible F (7, u) then 

(•, 7 v) G- authexec F (p , a, r, cr 0 ) 
assert v 

Z[t\ G- SUCCESS 

else 

r G- ABORT 
Z[f] -f- ABORT 

<p G- sign i (cOMMiT||t||u||i||Z[t]||fl'[t]) 
send message [COMMIT, u, t, Z[t], <p] to S 
u G- _L 

retum r // response of operation aip-invoke(o) 

upon recv. msg. [UPDATE-AUTH, o, r, 07, <p, q , 5 , a] from S' do 
assert verify^, cOMMiT||5f||o||*||Z[g]||if [g]) 

(og, j) <- (5 

(a, ip) G- a 

assert verify^(ip, auth||o,5||<2 — l|j-ff [g — l]||a) 
if Z[q\ = SUCCESS then 

(o', a' 0 , v) G- authexec F (o , a, r, 07) 

assert w 

else 

K, Co) G- (a, -L) 

^ si^/i^AUTHlIollgrll^^lla') 
send message [COMMIT-AUTH, a', cr^, 1//] to S' 
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Algorithm 2 ADS integrity protocol (AIP) for client C t , continued 


function extend-chain(o , l,j) 
if H[l\ = + then 

H[l] ^ hash(H[l - l}\\o\\l\\j) 

// extend by one 

else if H[l] ^ hash(H[l — l]||o |f||j) then 
return false 

// server replies are inconsistent 

return TRUE 

function check-view(6 , 6, a, ip) 
assert length(S) > 1 
if b = c then d <— c — 1 else d •*— c 
for k = 1, ..., length(S) do 
l i — d + k 

( 0,Z,(p,j) <T- S[k} 

assert extend-chain(o, l, j) 
assert verifyj(<p, commit \\l o j 7J[1]) 
assert verifyj(ip, auth o 6 \\H[b] a) 

// variables o and j keep their values 

c d + length{5) 

// all operations in 6 have been cleared 

function check-pendingloj) 
assert fength(w) > 1 
for k = 1 ,..., length(u>) do 
l t — c + k 
(O, T,j) Uj[k} 

assert extend-chain(o,l, j) A verify( t. INVOKE o j) 
assert o = u A j = i 

// variables o and j keep their values 


over t, the sequence number of o, its status, and its hash chain entry, sends a COMMIT message (with t, 
the operation, and the signature) to S, and outputs the response r. 

3.4.2 Passive phase 

The server stores the content of all incoming COMMIT messages in O and processes them in the order 
of their sequence numbers, indicated by 6. When an operation with sequence number 6+1 has been 
committed but not yet authenticated by the client and applied by S (i.e., upon 0[6+l] / + A A[b+ 1] = 
+), the server uses query F to compute the response r and to extract the auxiliary data a D from the current 
state s. It sends this in an UPDATE-AUTH message to C%, also including the operation at position 6 (from 
0[b]) and its authenticator (taken from ,4 71), which have been computed before. These values allow the 
client to verify the authenticity of the response for the operation at position 6+1. 

The client C t then receives this UPDATE-AUTH message (for o and sequence number q), and first 
validates the message contents. In particular, C t verifies that the authenticator a is covered by a valid 
AUTH-signature by client Cj with sequence number q — 1, using C+s hash chain entry H\q — 1]. 

Next, if o was not aborted, i.e., Z\q\ = SUCCESS, the client invokes authexecF to verify that the 
auxiliary data and the response are correct, and to generate new auxiliary data s' Q and a new authenti¬ 
cator a', which vouches for the correctness of the state updates induced by o. Otherwise, C, skips this 
step, as the authenticator does not change. Then Ci issues an AUTH-signature ip' and sends it back to S 
together with a 1 and s' Q in a COMMIT-AUTH message. 

As the last step in the passive phase, S increments 6, stores the data in the COMMIT-AUTH message 
at A[b], and if the operation did not abort, S applies it to s through refreshp. 
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Algorithm 3 ADS integrity protocol (AIP) for server S 

state 

t £ No: seqno. of last invoked op., initially 0 
b £ No: seqno. of last applied op., initially 0 
/ : N —> O x {0,1}* xN: invoked ops., initially empty 

O : N -> O x {0,1}* x Z x {0,1}* x N: committed ops., initially empty 

A : No —► {0,1}* x {0,1}*: authenticators, init. A [0] = ao 
s £ {0,1}*: state of the service, initially s = So 


upon receiving message [invoke, o, t , c] from Ci do 
t i — t - hi 

I[t] £- (o, r, i) 

if b = c then 6 <— (0[6]) else S <— ( 0[c + 1], ..., 0[b]} 
w t— (I[b + l],I[b + 2],..., /[t]) 

(p, •) <— separate-pending(i,cu) 

(r,cr 0 ) «- query F (s,/z) 

send message [REPLY, <5,6, A[b], w, t, r, <r 0 ] to Ci 

// all pending operations 

upon receiving message [COMMIT, o, q 1 z, <f>] from Ci do 

0[q} £- (o, z, </>, i) 


upon 0[b + 1] / 1 A A[b + 1] = _L do 
£- 0[b+ 1] 

if z = SUCCESS then 

(r,cr Q ) «- query F (s, o) 

else 

(r, do) £- (T, -L) 

send msg. [UPDATE-AUTH, o, r, tr 0 , </>,b+ 1, 0[b], A [6]] to Cj 


upon receiving message [COMMIT-AUTH, a, cr 0 , ip\ from U, do 
b <- b+ 1 

A[b\ <- {a, ip) 

(o, z,-,-) £- 0[b] 
if z = SUCCESS then 

s <— refreshp(s, o, a 0 ) 



3.5 Remarks 

As in BST lf34ll and in COP |j9], operations that do not interfere with each other may proceed without 
blocking. More precisely, if some pending operation is not compatible with the current operation, the 
latter is aborted and must be retried later. Preventing clients from blocking is highly desirable but cannot 
always be guaranteed without introducing aborts ifTOll . The potential for blocking has led other systems, 
including SPORC lfl3l and FAUST |j8|, to adopt weaker and less desirable guarantees than fork-linear- 
izability. 

Obviously, it makes no sense for a client to retry its operation while the non-compatible operation 
is still pending. However, the client does not know when the contending operation commits. Additional 
communication between the server and the clients could be introduced to signal this. Alternatively, the 
client may employ a probabilistic waiting strategy and retry after a random delay. 

In the following we assume that S is correct. The communication cost of AIP amounts to the five 
messages per operation. Every client eventually learns about all operations of all clients, as it must 
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cleai - them and include them in its hash chain. However, this occurs only when the client executes an 
operation (in REPLY). At all other times between operations, the client may be offline and inactive. 
In a system with n clients that performs h operations in total, BST ll 34 ll and COP | 9 j] require 0(nh) 
messages overall. AIP reduces this cost to 0(h) messages, which means that each client only processes 
a small constant number of messages per operation. 

The size of the INVOKE, COMMIT, UPDATE-AUTH, and COMMIT-AUTH messages does not depend 
on the number of clients and on the number operations they execute. The size of the REPLY message 
is influenced by the amount of contention, as it contains the pending operations. If one client is slow, 
the pending operations may grow with the number of further operations executed by other clients. Note 
that the oldest pending operation is the one at sequence number 6 + 1 ; hence, all operations ordered 
afterwards are treated as pending, even when they already have been committed. The REPLY message 
can easily be compressed to constant size, however, by omitting the pending operations that have already 
been sent in a previous message to the same client. See the protocol extensions in Sec. 3.7 for further 
discussion. 

The functionality-dependent cost, in terms of communicated state and auxiliary data, is directly 
related to the ADS for F. In practice, hierarchical authenticated search structures, such as hash trees 
and authenticated skip lists, permit small authenticators and auxiliary data |[T2l . 


3.6 Correctness 

We consider three cases: (1) S is correct and the clients execute operations (la) sequentially or (lb) 
concurrently; and (2) S is malicious. 

In case (la), all operations execute one after each other. When, furthermore, the COMMIT-AUTH 
message from a client reaches S before the next operation is invoked, then AIP is similar to “serialized” 
SUNDR |[23l and the “lock-step protocol” of Cachin j 6 j. This means that a client C, executing an 
operation o receives a REPLY message with all authenticated operations that exist in the system and 
only o as pending operation. Then 6 = 6 + 1, and later C, commits o and authenticates o without further 
operations intermixed at S nor at any client. Clearly, this execution is linearizable and satisfies the first 
condition of a fork-linearizable Byzantine emulation. 

In case (lb), there may exist pending-other operations, but since C t verifies that its own operations 
are compatible with them, the response value is correct. As S is correct, the views of all clients are 
equal, i.e., prefixes of each other, and this ensures linearizability. 

For case (2), note that every client C{ starts to extend its view from a cleared, authenticated operation 
and the corresponding signed authenticator a. If the pending-other operations in 7 are compatible with o 
and the response is valid w.r.t. a (according to authexeep), then it is safe for C) to output the response 
and thereby include it into its view. The malicious S may order the operations in 7 differently at other 
clients, creating a fork, but they can be omitted from the view of C t . The hash chain maintained by 
Ci contains a condensed representation of its entire view. By using its own hash chain entry during the 
verification of the COMMIT and AUTH signatures of other clients, Ci ensures that the views of these 
other clients are equal. Hence, whenever an operation o appears in the views of two clients, also their 
views are the same up to o. This ensures fork-linearizability. 

3.7 Extensions 

In order to keep the complexity of the protocol description for AIP at a comprehensible level, we present 
important efficiency improvements informally here. 
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Eliminating aborted operations. The first extension removes aborted operations from being consid¬ 
ered as pending. Recall that in AIP, an operation o is included with the pending operations until the 
executing client has authenticated o and the server has processed it during the respective background 
phase, even if the operation was aborted. This has the drawback that later operations may not be com¬ 
patible with o and abort unnecessarily. However, if o was aborted, this has been committed by the client, 
and S has received its COMMIT message, then S can include this with the list of pending operations sent 
for a later operation o'. The client that executes o' will take into account that o was aborted and ignore it 
for determining whether o' is compatible with the pending operations. Depending on the workload, this 
reduces further aborts. 

Batching and delegating operation authentication. Recall that the clients authenticate operations 
in the order of the server-assigned sequence numbers. Some client C s i ow may fall behind with this 
step, and when the other clients proceed faster and execute more operations, the number of pending 
operations grows continuously. This creates much more work for the faster clients for processing the 
REPLY message and slows them down. 

However, since all clients trust each other, one can modify AIP such that another client Cf ast may step 
in for C s i ow , handle the UPDATE-AUTH message, and sign the authenticator for the operation of C s i ow . 
Only small changes to the data structures arc needed to accommodate this change. Ideally Cf ast has 
more processing power or is closer to S on the network than C s i ow : this choice should be determined 
heuristically based on the measured performance or network delays. 

Extending the above idea, S may actually batch all non-authenticated operations when an operation 
from C\ commits at sequence number q. Hence, S sends the UPDATE-AUTH messages for all operations 
between b and q to C, and delegates the step of authenticating them to C t . This works because a client 
can authenticate two consecutive operations without going back to S. The server may then record the 
COMMIT-AUTH responses from the fastest client and inform the others that their help with authenticating 
operations is no longer needed. 

Passive phase only for update operations. Recall that the protocol executes a functionality F whose 
operations can be separated into query and update operations (Q and U, respectively). Queries do not 
change the state; as is easy to see, they are compatible with every subsequent operation. But the passive 
phase of AIP is only needed for creating a new authenticator, after the state has changed. Therefore we 
can eliminate the passive phase for all query operations; this considerably improves the efficiency of the 
protocol for read-intensive applications. 

In particular, for every operation o € Q, the passive phase is skipped and the verification operations 
are adjusted accordingly. When a client C, has committed a query operation, the server immediately 
“applies” it and does not send an UPDATE-AUTH message later. For implementing this, the sever has to 
maintain another variable d with the sequence number of the operation that most recently modified the 
state. The REPLY and UPDATE-AUTH messages now contain the operation 0[d] that allows clients to 
verify the corresponding authenticator A[d\. 

Tolerate client crashes. Crashes of clients are not included in the formal model here, but must be 
considered in practice. In order to tolerate crashes, we reuse the second extension above. Since a 
client Cfaii may crash before it finishes both protocol phases, the server will never receive the missing 
message(s) and the operation is never authenticated. Going beyond our formal model, suppose that 
another client C, has determined that Cf a u has crashed. Then C, may take over, abort the hanging 
operations, and commit and authenticated in place of the failed client. It is important that Cf a n must 
not execute any operations again later, hence the failure detection should be reliable; if Cr a n may join 
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again later, it must be given a new identity. Group management protocols for adding and removing 
clients dynamically have been discussed in the context of existing systems, such as VENUS li30l and 
SPORC OED. 

4 Verification of integrity and consistency of cloud object storage (VI¬ 
COS) 

We are now ready to introduce our main contribution, the protocol for verifying the integrity and con¬ 
sistency of cloud object storage, abbreviated VICOS. It leverages AIP from the previous section and 
provides a fork-linearizable Byzantine emulation for a practical object-store service, in a manner that 
is transparent to the storage provider. We first define the operations of the cloud storage service and 
outline the architecture of VICOS. Next we instantiate AIP for verifying the integrity of a simple object 
store and show how VICOS extends this to practical cloud storage. 

More precisely, VICOS consists of the following components (see Fig. [2]): 

1. A cloud object store (COS) service with a key-value store interface, as offered by commercial 
providers. It maintains the object data stored by the clients using VICOS. 

2. An AIP client and an AIP server, which implement the protocol from the previous section for 
the functionality of an authenticated dictionary (ADICT) and authenticate the objects at the cloud 
object store. The AIP server runs remotely as a cloud service accessed by the AIP client. This is 
abbreviated as AIP with ADICT. 

3. The VICOS client exposes a cloud object store interface to the client application and transparently 
performs integrity and consistency verification. During each operation, the client consults the 
cloud object store for the object data itself and the AIP server for integrity-specific metadata. In 
particular, AIP server running ADICT stores a cryptographic hash of every object. 

Note that the cloud object store as well as the AIP server are in the untrusted domain; they may, in fact, 
collude together against the clients. 



Figure 2. Architecture of VICOS: the two untrusted components of the cloud service are shown at the 
top, the trusted client is at the bottom. 
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4.1 Cloud object store (COS) 


The cloud object store is modeled by key-value store (KVS) and provides a “simple” storage service to 
multiple clients. It stores a practically unbounded number of objects in a flat namespace, where each 
object is an arbitrary sequence of bytes (or a “blob,” a binary large object), identified by a unique name 
or key. We assume that clients may only read and write entire objects, i.e., it is not possible to read from 
or write into the middle of an object, as in a file system. 

Our formal notion of a KVS internally maintains a map M that stores the values in V under their 
respective keys taken from a universe 1C. It provides four operations: 

1. kvs-put(k, v): Stores a value v £ V under key k £ 1C, that is, M[k ] v. 

2. kvs-get(k): Returns the value stored under key k £ 1C, that is, M[k\. 

3. kvs-del(k): Deletes the value stored under key k £ 1C, that is, M[k\ £- _L. 

4. kvs-list( ): Returns a list of all keys for which a value is stored, that is, the list (k £ K\M[k] ^ _L). 

This API forms the core of many real-world cloud storage services, such as Amazon S3 or OpenStack 
Swift. Typically there is a bound on the length of the keys, such as a few hundred bytes, but the stored 
values can be much larger and practically unbounded (on the order of several Gigabytes). For simplicity, 
we assume that the object store provides atomic semantics during concurrent access, being aware that 
cloud storage systems may only be eventually consistent |4} due to network partitions. 

Many practical cloud object stores support a single-level hierarchical name space, formed by con¬ 
tainers or buckets. We abstract this separation into the keys here; however, a production-grade system 
would introduce this separation again by applying the design of VICOS for every container. 

4.2 Authenticated dictionary implementation (ADICT) 

VICOS instantiates AIP with the functionality of a KVS that stores only short values. In order to 
distinguish it from the cloud object store, we refer to it as authenticated dictionary, denoted by ADICT, 
with operations adict-put, adict-get, adict-del, and adict-list. 

The implementation of ADICT uses the well-known approach of building a hash tree over its en¬ 
tries 13 Until; see Alg.[4]-[5]for the details of how ADICT is implemented within AIP. The AIP server 
stores the values in a map D and maintains a hash tree T, constructed over the list of key-value pairs 
stored in the map, according to a fixed sort order on the keys. That is, every leaf node of the hash tree 
is computed by hashing the node key, its value, and the key of the successor leaf node together. The 
next node has to be included in order to authenticate the absence of a key in response to a query for a 
non-existing key (e.g., 1281 1. The root of the hash tree serves as the authenticator for ADICT. 

For the adict-put, adict-get, and adict-del operations, the server extracts those paths from T that are 
necessary to verify the correctness of the retrieved value and places them in s Q . For adict-put and adict- 
del operations, query ADICT also places these paths into s 0 because the client needs them to construct 
the updated root hash. For adict-list, the complete hash tree is included in s Q . The asterisks (*) in 
Alg.[4]-[5] denote some additional data and steps necessary to verify the predecessor or successor leaves 
for authenticating an absent key (details of this are omitted here and can be found in the literature fl2l ). 

The compatible ADICT (p, u ) function of Alg.[4]-[5]defines the compatibility relation among the oper¬ 
ations of the authenticated dictionary; VICOS supports the same KVS interface and inherits this notion 
of compatibility for the cloud-storage operations. For more general services like databases, one would 
invoke a transaction manager here. For ADICT, the compatibility between a first (pending) operation u 
and a second (current) operation o is given by Table [T] For instance, adict-put followed by adict-get for 
the same key or followed by adict-list are not compatible, whereas two adict-list and adict-get opera¬ 
tions are always compatible. 
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Algorithm 4 Authenticated dictionary implementation (ADICT) for AIP, Part 1 
state 

D : K, —> {0,1}*: authenticated dictionary, initially empty 
T: hash tree over D, initially empty 

function query adict ((D , T), o) 

if o = adict-put(k, d)Vo = adict-del(k) then 
r <— 1 

s 0 t— sibling nodes on path (*) from k to root in T 
else if o = adict-get(k) then 

r <— D[k\ 

s 0 t— sibling nodes on path (*) from k to root in T 
else // o = a dict-list() 

r^{kG IC\D[k] ± _L) 

s 0 r 

return (r, s 0 ) 

function authexec AD icT(o, a, r , s Q ) 

if o = adict-put(k , v) V o = adict-get(k) V o = adict-del(k) then 
if s 0 is not a valid path (*) from k to tree root a then 
return (•, •, false) 
if o = adict-put(k, v) then 

insert leaf node k with value v in the tree 
s ' 0 t— updated path from k to nee root 
a't— updated hash-tree root 
else if o = adict-get(k) then 

if path (*) not consistent with node k holding r then 
return (•, •, false) 

s o -L 

a' <— a 

else if o = adict-del(k) then 

delete leaf node k from the tree 
s ' 0 t— updated paths from siblings of k to tree root 
a't— updated hash-tree root 
else // o = adict-listQ 

if ?’ is not list of keys in leaves of nee with root a then 
return (•, •, false) 
s o -L 
o! <— a 

return (a', s’ 0 , true) 


The advantage of considering operation compatibility over commutativity (as used in ACOP f9|, for 
instance) becomes apparent here: only 8 pairs among the 49 cases shown are not compatible, whereas 
22 among 49 cases do not commute and would be aborted with commutativity. 
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Algorithm 5 Authenticated dictionary implementation (ADICT) for AIP, Part 2 

function refresh A i)ici((D■ T),o , s a ) 
if o = adict-put(k, v) then 
D[k] <r~ V 

update path in T from k to root, as taken from s 0 
else if o = adict-del(k) then 

D[k] <- X 

update path in T from k to root, as taken from s Q 

return (D, T) 

function compatible ADICT (p, u) 

for o € p do 

if -^compatible ADICT {u, o) then // See Table[l]for the compatible ADICT 0 relation on operations 

return false 
return TRUE 
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V 

V 

V 

V 

V 
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Table 1. The compatible ADICT (-, ■) relation for ADICT and the KVS interface, where x, y E 1C denote 
distinct keys 
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4.3 VICOS client implementation 


VICOS emulates the key-value store API of a cloud object store (COS) to the client and transparently 
adds integrity and consistency verification. As with AIP, consistency or data integrity violations com¬ 
mitted by the server are detected through assert; any failing assertion triggers an alarm. It must be 
followed by a recovery action whose details go beyond the scope of this paper. Analogously to AIP, 
VICOS may return ABORT; this means that the operation was not executed and the client should retry it. 

Algorithm 6 Implementation of VICOS at the client. 

function vicos-put{k , v) 
x 4— a random nonce 
cos-put(k\\x, v) 
h 4— hash(v) 

r 4— aip-invoke(adict-put(k, ( x , h))) 
if r = abort then 
cos-del(k\\x) 

return r 

// concurrent incompatible operation 

function vicos-get(k) 

r 4— aip-invoke(adict-get(k)) 
if r = abort then 
return abort 
(. x , h) 4— r 
v 4- cos-get(k\\x) 
assert hash(v) = h 
return v 

// concurrent incompatible operation 

function vicos-del(k) 

r 4— aip-invoke(adict-del(k)) 
if r = abort then 

return abort 
cos-dei(fc||*) 

return r 

// concurrent incompatible operation 

// deletes all keys with prefix k 

function vicos-list() 

r 4— aip-invoke(adict-listQ) 
if r = abort then 
return abort 

return r 

// concurrent incompatible operation 


Algorithm [6] presents the pseudo code of the VICOS client. Basically, it protects every object in 
the COS by storing its cryptographic hash in the authenticated dictionary (ADICT). Operations on the 
object store trigger corresponding operations on COS and on ADICT, as provided by AIP for consistency 
enforcement. 

In order to prevent race conditions, VICOS does not store the hash of an object under the object’s key 
in COS directly, but translates every object key to a unique key for COS. Otherwise, two concurrent op¬ 
erations accessing the same object might interfere with each other and leave the system in an inconsistent 
state. More precisely, in a vicos-put(k , v) operation, the client chooses a nonce x (a value guaranteed to 
be unique in the system, such as a random string) and stores v in COS using cos-put(k\\x, v) with the 
translated key k\\x. Furthermore, it computes h 4— luish(v) and stores (x. h) in AD using adict-put at 
key k. The cos-put and cos-get operations actually stream long values. When adict-put aborts due to 
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concurrent operations, the client deletes v again from COS using co.s-de/(/;j|.x'). 

For a vicos-get(k) operation, the client first calls adict-get(k) and retrieves (x, h). Unless this 
operation aborts, the client translates the key and calls cos-get(k’||x) to retrieve the value v. After v 
has been read (or streamed), the client compares its hash value to h, asserts that they match, and then 
outputs v. 

Without key translation, two concurrent vicos-put operations o\ and 02 writing different values to 
the same key k might both succeed with cos-put(k. v\) and cos-put(k, vo), respectively, but the adict-put 
for 02 might abort due to another concurrent operation. Then COS might store but ADICT stores the 
hash of v 1 and readers would observe a false integrity violation. Thanks to key translation, no versioning 
conflicts arise in the COS. Atomicity for multiple operations on the same object key follows from the 
properties of AIP with the ADICT implementation. The vicos-dcl(k) and vicos-list( ) operations proceed 
analogously; but the latter does not access COS. 

4.4 Correctness 

It is easy to see that the implementation of VICOS satisfies the two properties of a fork-linearizable 
Byzantine emulation. First, when S is correct, then the clients proceed with their operations and all 
verification steps succeed. Hence, VICOS produces a linearizable execution. The linearization order 
is established by AIP running ADICT. Furthermore, when the clients execute sequentially, then by the 
corresponding property of AIP, no client ever receives ABORT from ADICT. 

Second, consider the case of a malicious server controlling COS and the AIP server together. AIP 
ensures that the operations on ADICT ( adict-put , adict-get, etc.) are fork-linearizable according to 
Sec. [4] The implementation of ADICT follows the known approach of memory checking with hash 
trees |j5][28] and therefore authenticates the object hash values that VICOS writes to ADICT. According 
to the properties of the hash function, the object data is uniquely represented by its hash value. Since 
VICOS ensures the object data written to COS or returned to the client corresponds to the hash value 
stored in ADICT, it follows that all operations of VICOS are also fork-linearizable. 

5 Prototype 

We have implemented a prototype of VICOS in Java; it consists of a client-side library (“VICOS client”) 
and the server code (“VICOS server”). The system can be integrated with applications that require 
cryptographic integrity and consistency guarantees for data in untrusted cloud storage services. It is 
available as open-source on GitHub (https : //github . com/ibm-research/vicos). 

5.1 VICOS implementation 

The client-side library uses the BlobStore interface of Apache jclouds (Version 2.0.0 -Snapshot, https : 
// j clouds . apache . org/ ) for connecting to different cloud object stores. The server runs as a 
standalone web service, communicating with the client-side library using the Akka framework (Version 
2.4.4, http : / /www. akka . io). 

A kk a is an event-driven framework which supports the actor model (20]]. It fits perfectly into the 
system model of VICOS and allows for rapid development. This simplifies the actual protocol im¬ 
plementation because of the high level of abstraction, especially relating to network operations and 
concurrency. 

The client library as well as the server code are implemented as actors within the framework. Actors 
are independent units which can only communicate by exchanging messages. Every actor has a mailbox 
that buffers all incoming messages. By default messages are processed in FIFO order by the actor. This 
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allows the server protocol implementation to process all incoming messages sequentially and execute 
each protocol step atomically, that is, mutually exclusive with respect to all others. The implementation 
of VICOS therefore closely follows the high-level description according to Sec. [3] Note that this clearly 
limits the systems performance by not utilizing modem multi-core architecture, on the other hand. 

We developed the VICOS client library so that it may easily be integrated into existing applications 
to provide integrity protection. It uses the modular approach of AIP instantiated with an authenticated 
data structure (ADS). A developer only needs to implement the desired functionality, by defining the 
state (e.g., the internal KVS data structure), a set of operations, and their compatibility relation. For 
that reason, we have defined two interfaces: state and operation processor. Operations are described us¬ 
ing Google’s Protocol Buffers (https : / /developers . google . com/protocol-buffers/) 
executed by the operation processor. This modular concept allows us to reuse and extend the core 
implementation of the protocol. 

The VICOS prototype provides a simple KVS functionality by implementing these interfaces. More 
precisely, the KVS state is a map supporting GET, PUT, DEL and LIST operations. The KVS opera¬ 
tion processor provides the implementations of query, authexec, refresh, and compatible as described in 
Alg.ffl The client and the server protocol each contain an instance of the KVS operation processor. 
The client exposes a simple KVS interface, adding Java Exceptions for signaling integrity and consis¬ 
tency violations. Furthermore, the client library is completely asynchronous and supports processing 
the AIP passive phase in the background without blocking the client process. 

The cryptographic signatures can be implemented in multiple ways. According to the security as¬ 
sumptions of the model, all clients trust each other, the server alone may act maliciously, and only 
clients issue digital signatures. Therefore, one can also adopt a simplified trust model with “signatures” 
provided by a message-authentication code (MAC). For many applications, where strong mutual trust 
exists among the clients, MACs suffice and will result in faster execution. On the other hand, this 
simplification renders the system more fragile and exposes it easily to attacks by clients. 

In particular, VICOS uses HMAC-SHA1 with 128-bit keys provided by the Java Cryptography 
Extension as the default signature implementation. The code also supports RSA and DSA signatures 
with 2048-bit keys. A user can choose between these signature implementations via a configuration 
file. This approach also allows developers to change to a different signature implementation, or even 
implement their own according to new requirements. Particularly, this might be useful for porting the 
code to other platforms such as mobile devices, with less computation power. 

The core implementation of VICOS consists of ~3400 sloe, the server part is ~400 sloe more, 
whereas the client part including the integration with the evaluation platform (see below) takes ~800 sloe 
extra. 

5.2 Practical issues and optimizations 

While developing VICOS and experimenting with it, we obtained experience with Akka and gained 
insight into the protocol’s operation itself. This has led to further optimizations described here. 

Bounded pending list. An issue that we discovered in A kk a while implementing VICOS is Akka’s 
default maximum message size of only 128 kB. In particular, this becomes a problem in VICOS when 
REPLY messages include a large partial state or a very long list of pending operations. By configuring 
Akka with larger message sizes (we used a maximum size of 128 MB), the direct limitation disappears. 

However, we experienced that large messages impact the overall performance negatively. When the 
number of pending operations increases, the resulting very large messages slow down the operations of 
VICOS. Therefore, we implemented a way to bound the length of the pending-operations list, that is, we 
introduced a maximum number of pending operations as a configurable value and modified the protocol. 
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When this maximum is reached, the server buffers all new incoming requests (INVOKE messages) until 
enough other operations have completed and the number of pending operations goes below the limit. 
We tested with different maximum sizes for the pending list from 32 up to 1024 operations, and chose a 
limit of 128 for the evaluations. 

A more robust solution of that issue would be to signal the clients to wait before sending more re¬ 
quests, instead of just buffering them at the server. Although limiting the number of pending operations 
under high server load increases the latency of client requests, it also increases the overall performance 
and stability of the system. In summary we found that the benefits of this optimization outweigh its 
drawbacks. 

Message delivery order. During development we experienced slow performance caused by the FIFO 
order in which the protocol actors processed the arriving messages. Therefore, we implemented priority 
mailboxes for the server and the client actor and defined a priority rule to prefer COMMIT, UPDATE- 
AUTH, COMMIT-AUTH messages over INVOKE and REPLY messages. This has the immediate benefit 
that the server processes UPDATE-AUTH messages and thereby completes the PASSIVE PHASE of already 
authenticated operations before it starts working on new INVOKE messages. This preference shortens 
the list of pending operations directly. Certainly, this may increase the response time for new operations 
again, but eventually, it prevents more operations from aborting due to conflicts under high load. 

6 Evaluation 

This section reports on performance measurements with the VICOS prototype. They study the gen¬ 
eral overhead of integrity protection, the scalability of the protocol, and the effect of (in-)compatible 
operations. 

6.1 Experimental setup 

The experiments use cloud servers and an OpenStack Swift-based object storage service (http: // 
swift. openstack . org) of a major cloud provider with about two dozen data centers world-wide 
(Softlayer — an IBM Company, http: / / www. soft layer . com/object-storage). 

The VICOS server runs on a dedicated “baremetal” cloud server with a 3.5GHz Intel Xeon-Haswell 
(E3-1270-V3-Quadcore) CPU, 8 GB DDR3 RAM, and a 1 Gbps network connection. The clients run on 
six baremetal servers in total, each server with 2x 2GHz Intel Xeon-SandyBridge (E5-2620-HexCore) 
CPUs, 16 GB DD3 RAM, and a 1 Gbps network connection. All clients are hosted in the same data 
center. All machines run Ubuntu 14.04-64 Linux and Oracle Java (JRE 8, build 1.8.0_77-b03). 

To simulate a realistic environment, we conduct experiments in two settings as shows in Table [2] 
A datacenter setting, with all components in the same data center (“Amsterdam”), and a wide-area 
setting, where the VICOS server and COS are located together in one data center (“Milan”), and the 
clients at a remote site (“Amsterdam”). 


Setting 

Clients 

VICOS server 

Cloud object storage 

Latency [ms] 

Datacenter 

Amsterdam 

Amsterdam 

Amsterdam 

< 1 

Wide-area 

Amsterdam 

Milan 

Milan 

~ 10 


Table 2. Evaluation setting 
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The datacenter setting establishes a best-case baseline due to the very low network latencies (< 1 ms). 
This deployment is not very realistic in terms of the security model because the clients and the storage 
service are co-located. 

The wide-area setting exhibits a moderate network latency (round-trip delay time of ~20 ms) be¬ 
tween the two data centers and models the typical case of geographically distributed clients accessing a 
cloud service with its point-of-access on the same continent but in different countries. 



Figure 3. The experimental setup, with one COSBench controller and many COSBench drivers, access¬ 
ing the cloud storage service through the VICOS client. 

The evaluation is driven by COSBench (Version 0.4.2 -https : //github . com/intel-cloud/ 
cosbench), an extensible tool for benchmarking cloud object stores. We have created an adapter to 
drive VICOS from COSBench, as shown in Fig. [3] COSBench uses a distributed architecture, consist¬ 
ing of multiple drivers, which generate the workload and simulate many clients invoking concurrent 
operations on a cloud object store, and one controller, which controls the drivers, selects the workload 
parameters, collects results, and outputs aggregated statistics. In particular, the COSBench setup for 
VICOS reports the operation latency, defined as the time that an operation takes from invocation to 
completion, and the aggregated throughput, defined as the data rate between all clients and the cloud 
storage service. 

COSBench invokes “read” and “write” operations, implemented by vicos-get and vicos-put, respec¬ 
tively. Every reported data point involves read and write operations taken over a period of at least 30 s 
after a 30 s warm-up. In the experiments two configurations are measured: 

1. The native object storage service as a baseline, with direct unprotected access from COSBench to 
cloud storage, but accessing the cloud storage through the jclouds interface; and 

2. VICOS, running all operations from COSBench through jclouds and the verification the protocol. 

6.2 Results 

6.2.1 Cryptography microbenchmark 

In a first experiment we study how different signature implementation affect the computation and net¬ 
work overhead of VICOS. We implemented digital signatures using RSA and DSA with 2048 bit keys, 
and additionally HMAC-SHA1 with 128 bit keys. The cryptographic algorithms are provided by the 
SunJCE version 1.8 provider. 
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We measured the time it takes on a client to sign and verify an invoke message using the three sig¬ 
nature implementations. Figure [4] shows that the RSA signing takes around 5 ms, while the verification 
takes around 220 ps. DSA takes around 4 ms for signing and around 1.5 ms for verification, whereas 
HMAC signing and verification only takes less than 20 ps. Additionally, the resulting signature sizes 
have a direct effect on the message sizes and the network load. RSA signatures are 256 byte, while DSA 
signatures are only 40 byte. F1MAC reduces the signature size to 20 byte. 

We use HMAC-SF1A1 as the default implementation of signatures in the remainder of the evaluation. 
Its operations are much faster than for RSA and DSA signatures, reducing the computation overhead 
at the client. Moreover, the smaller signature size of F1MAC-SHA1 also reduces the network overhead 
of VICOS. 



HMAC RSA DSA 

Signature implementation 


Operation 

Sign 
Verify 


■ 


Figure 4. The average time for digital-signature operations (HMAC, RSA, and DSA); note that the 
y-axis uses log-scale. 


6.2.2 Object size 

In this experiment we study how the object size affects the latency and the throughput of VICOS. We 
define a workload with a single client executing read and write operations for objects of size 1 kB, 10 kB, 
100kB, and 1MB. 

Fig. [5] shows that the latency and throughput of VICOS behave very similar to the native system. 
As expected, we observe that VICOS introduces an overhead that incurs a small cost compared to 
unprotected access to storage. In particular, for the datacenter setting, VICOS increases latency by an 
average of 16.2% for read, and 24.0% for write', it decreases throughput by an average of 15.8% for 
read, and 17.7% for write. We also expect the overhead to decrease with bigger objects. However, we 
could not find this effect in the datacenter setting: here the overhead remains practically constant, from 
the small to the large objects. In the wide-area setting, the overhead is approximately the same for the 
smaller objects (1 kB and 10 kB), but it indeed decreases as the object size grows and disappears at the 
largest object size (1000 kB). 

Interestingly, in the wide-area setting, the relative performance in terms of throughput is reversed be¬ 
tween read and write for 1000 kB objects. Whereas read has lower latency and achieves better through¬ 
put than write in all other experiments, this relation is reversed in the right-most data points of Fig. [5] 
This may be caused by caching data on the cloud object store, which improves read performance for 
smaller objects, but disappears when larger objects are stored and accessed less frequently than in the 
datacenter setting. 
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Figure 5. The effect of different object sizes: Latency and throughput of read and write operations with 
one client. 


6.2.3 Number of clients 

We also study the scalability of VICOS by increasing the number of clients. The workload uses up 
to 128 clients (spread uniformly over the six COSBench drivers), and 64 objects with a fixed size 
of 10 kB. One half of the objects are designated for read operations and the other half for write op¬ 
erations, respectively. This division prevents concurrency conflicts among the client operations. As 
mentioned in Sec. |5.2| we restrict the size of the pending list in the protocol to 128 operations. We 
do not use a large number of clients because of the underlying assumption that clients trust each other, 
which might not be realistic in much larger groups. 

As Fig. [6] shows, the native system throughput scales linearly until the network is saturated with 64 
clients in the data center setting. VICOS follows the same behavior but reaches saturation already 
with 32 clients. In contrast, in the wide-area setting, VICOS becomes saturated with 8 clients, from 
where throughput remains almost constant and latency grows. No saturation is evident with the native 
configuration and up to 128 clients. 

The reason for the slower operation in the wide-area experiment is that all requests of the clients 
are handled by the VICOS server sequentially and thus it becomes a bottleneck of the system. Due to 
the higher latency in the wide-area setting, operations remain longer in the pending queue. This means 
more work for the clients and the server. Since the active and passive protocol phases are asynchronous, 
clients may invoke the next operation already before they have completed a previous operation. Hence, 
they reach limit of the bounded pending queue and the throughput of VICOS remains limited at the rate 
imposed by the server’s operation. 
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Figure 6. Scalability with the number of clients: Latency and throughput of read and write operations 
with 10 kB objects. 


6.2.4 Concurrent operations 


Finally, we investigate the effect of conflicting concurrent operations. VICOS aborts an operation if 


it is not compatible with one of the pending operations (according to Sec. 2.4). In that case the client 
has to retry later. Protocols like BST ||34l and ACOP 0 are more cautious and abort as soon as two 
operations do not commute, which occurs more often. Hence, we define ACOP as our baseline for 
this experiment. The implementation throws an abort exception when a conflict occurs; that causes 
COSBench to report the operation as failed and to continue immediately with the next operation. At 
the end of each experiment COSBench reports the overall operation success rate. We expect a higher 


success rate for VICOS compared to ACOP as already discussed in Sec. 4.2 


To evaluate this behavior we created a workload with sixteen clients each invoking read and write 
operations over 64 objects with a fixed size of 10 kB. Object accesses are chosen according to “Zipf’s 
law”, which approximates many types of data series found in natural and social structures. The Zipf 
distribution is based on a ranking of the elements in a universe and postulates that the frequency of any 
element is inversely proportional to its rank in the frequency table. Thus, the most “popular” object will 
occur approximately twice as often as the second most popular one, three times as often as the third 
most popular and so on. Zipf distributions are often observed when users access websites ||T|. Figure [7] 
shows the object access rate using four different values for the Zipf factor 9. For 6 = 0.99 the access 
rate for the first object (with the biggest contention) is about 20%, and the least often accessed object is 
selected only with probability about 0.3%. With 9 = 0 the Zipf distribution corresponds to uniformly 
random access over the objects. 

Since COSBench does not support Zipf distributions by default, we implemented the algorithm 
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Figure 7. The effect of different values for the Zipf distribution factor 9 of the COSBench object 
selector: Selection rate for each object with 10000 selections, 64 objects, and varying 9. 
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Figure 8. The effect of conflicting concurrent operations: Success rate of read and write operations with 
10 kB objects, sixteen clients, and varying Zipf distribution factors 9. 


of Gray et al. fl8l for generating a Zipf-like access distribution, as also used in YCSB (https: // 
github. com/brianf rankcooper/YCSB). With this workload we cause operations to conflict 
by progressively increasing 9 6 {0, 0.5, 0.75, 0.99}. The higher the Zipf factor 9, the more clients 
concurrently access the same objects and invoke non-compatible operations, causing aborts. 

Figure [8] shows the operation success rates for VICOS and for ACOP, using the four different Zipf 
factors. Recall from Sec. |2.4| and from Table |T] that a put operation in the KVS interface is always 
compatible with every preceding operation and never aborts. Therefore, the write operations show 
100% success rate for VICOS. For the commuting operations in ACOP, on the other hand, writes are 
progressively more often aborted with increasing 9. The behavior of reads is similar for VICOS and 
ACOP because preceding writes cause aborts equally often. 

The advantage of VICOS over protocols considering only operation commutativity becomes evident 
here, in that write operations always succeed, and overall the abort rate is significantly reduced. 

6.2.5 Summary 

The performance evaluation shows that VICOS achieves its goal of adding consistency and integrity 
protection while remaining almost transparent to clients using cloud object stores. The cost added over 
the raw performance is most visible in the datacenter setting, which is not a realistic deployment for 
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the intended applications. Still this overhead remains limited to about 20% for accesses with high 
throughput (Sec. 6.2.2). With many clients performing operations concurrently, the extra cost may 
become noticeable (Sec. |6. 2.3 >, and further work is needed for decreasing this. It should be added that 
the VICOS prototype is currently a proof of concept and not product-level code. 


7 Related Work 

Many previous systems providing data integrity rely on trusted components. Distributed file systems 
with cryptographic protection provide stronger notions of integrity and consistency than given by VI¬ 
COS; there are many examples for this, from early research prototypes like FARSITE |[2j or SiR- 
iUS flT4l to production file-systems today (e.g., IBM Spectrum Scale, http : //www-03 . ibm. com/ 
systems/storage/spectrum/scale/). However, they rely on trusted directory services for 
freshness. Such a trusted coordinator is often missing or considered to be impractical. Iris liTil re¬ 
lies on a trusted gateway appliance, which mediates all requests between the clients and the untrusted 
cloud storage. Several recent systems ensure data integrity with the help of trusted hardware, such as 
CATS f35ll . which offers accountability based on an immutable public publishing medium, or A2M ifTTI . 
which assumes an append-only memory. They all require some form of global synchronization, usually 
done by the trusted component, for critical metadata to ensure linearizability. In the absence of such 
communication, as assumed here, they cannot protect consistency and prevent replay attacks. 

In CloudProof E9l . an object-storage protection system with accountable and proof-based data 
integrity and consistency support, clients may verify the freshness of returned objects with the help of 
the data owner. Its auditing operation works in epochs and verifies operations on one object only with a 
certain probability and only at the end of an epoch. Moreover, the clients need to communicate directly 
with the owner of an object for establishing integrity and consistency. 

Cryptographic integrity guarantees are of increasing interest for many diverse domains: Verena Ell , 
for example, is a recent enhancement for web applications that involve database queries and updates by 
multiple clients. It targets a patient database holding diagnostic data and treatment information. In 
contrast to VICOS, however, it relies on a trusted server that supplies hash values of data objects to 
clients during every operation. 

The remainder of this section discusses related work without trusted components for synchroniza¬ 
tion. With only one client, the classic solution for memory checking by Blum et al. [|5] provides data 
integrity through a hash tree and by storing its root at the client. Many systems have exemplified this 
approach for remote file systems and for cloud storage (e.g., Athos |[T6l0 . 

With authenticated data structures E8lf25l , the single-writer, multi-reader model of remote storage 
can be authenticated, assuming there is a trusted and timely way to distribute authenticators from the 
writer to all readers. In practice, this approach is often taken for software distribution, where new 
releases are posted to a repository and authenticated by broadcasting hash values of the packages over a 
mailing list. AIP as introduced in Sec. [^represents one way to generalize ADS for multiple writers. 

In the multi-client model, Mazieres et al. E6l[23 ] have introduced the notion of fork-linearizability 
and implemented SUNDR, the first system to guarantee fork-linearizable views to all clients. It detects 
integrity and consistency violations among all clients that become aware of each other’s operations. The 
SUNDR system uses messages of size D(n 2 ) for n clients iflOl . which might be expensive. The SUNDR 
prototype E3l description also claims to handle multiple files and directory trees; however, the protocol 
description and guarantees are stated informally only, so that it remains unclear whether it achieves 
fork-linearizability under all circumstances. 

As mentioned in Sec. [T] several systems have expanded the guarantees of fork-linearizability to 
different applications ltl3l and improved the general efficiency of protocols for achieving it lilOl . Others 
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have explored aborting operations lf24ll or introduced weak fork-linearizability in order to avoid blocking 
operations. In particular, SPORC lfl3l . FAUST [[8], and Venus ll30l sacrifice full linearizability to avoid 
aborts and blocking, respectively, and achieve weak fork-linearizability instead. The latter is a relaxation 
of fork-linearizability in which the most recent operation of a client may violate atomicity. 

The SPORC system fl3l is a groupware collaboration service whose operations may conflict with 
each other, but can be made to commute by applying a specific technique called “operational trans¬ 
formations.” Through this mechanism, different execution orders still converge to the same state; still 
SPORC achieves only weak fork-linearizability. 

Furthermore, VICOS also reduces the communication overhead compared to past systems consider¬ 
ably, since SUNDR, FAUST, and Venus all use messages of size 0(n) or more with n clients, whereas 
the message size in VICOS does not depend on n. 

The BST protocol lf34l supports an encrypted remote database hosted by an untrusted server that is 
accessed by multiple clients. Its consistency checking algorithm allows some commuting client opera¬ 
tions to proceed concurrently; COP and ACOP 0 extend BST and also guarantee fork-linearizability 
for arbitrary services run by a Byzantine server, going beyond data storage services, and support wait- 
freedom for commuting operations. VICOS builds directly on COP, but improves the efficiency by 
avoiding the local state copies at clients and by reducing the computation and communication overhead. 
The main advantage is that clients can remain offline between executing operations without stalling the 
protocol. 

8 Conclusion 

This paper has presented VICOS, a complete system for protecting the integrity and consistency of 
data outsourced to untrusted commodity cloud object stores. It shows, for the first time, how to realize 
multi-client integrity protection for generic functions with an authenticated data structure (ADS). Its 
two-phase protocol structure reduces the communication overhead compared to previous algorithms. 

VICOS works with commodity cloud storage services and ensures the best possible consistency 
notion of fork-linearizability. It supports wait-free client operations and does not require any additional 
trusted components. 

There are several challenges that this paper not address, which remain open for future work. An 
interesting question, for instance, is how to recover from an integrity violation. Since we assume only 
a single untrusted server and that client data resides at the cloud storage service, orthogonal techniques 
are needed for resilience of the data itself. 

Another interesting challenge would be to consider malicious clients, as one further step towards a 
more realistic system. For small groups of clients our system model makes sense, but for groups with 
hundreds of clients it seems difficult to maintain this assumption. The situation is especially interesting 
when a client colludes with the malicious server. 

Finally, the approach of AIP also be applied to services beyond cloud storage; for example, cloud 
and NoSQL databases, interactions in a social network, or certificate and key management services. 
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